Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract by RhysSullivan · Pull Request #992 · UsefulSoftwareCo/executor

RhysSullivan · 2026-06-12T22:43:34Z

Why

Expected tool failures (ToolResult.fail) resolve through the Effect success channel by design, which meant the tracer recorded healthy spans while callers were hitting upstream error walls. Auditing a week of traces showed three concrete gaps:

plugin.openapi.invoke spans never carried http.status_code — the annotation ran inside the inner OpenApi.invoke span (created by Effect.fn), not the wrapper span queries target. 0 of ~19.5k spans had it.
The majority of executor.tool.execute error spans had an empty status.message: most SDK tagged errors define no message field, and TaggedErrorClass instances default to "".
Some error spans rendered [object Object] as their only label (non-Error causes stringified with String(...)).

The absence of error data is indistinguishable from health, so none of this was noticed until incidents were debugged by hand.

What

Outcome attributes on tool spans. New annotateToolResultOutcome (exported from the SDK) stamps executor.tool.outcome (ok/fail), executor.tool.error_code, and executor.tool.error_status onto executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult. Codes and statuses are enumerable identifiers, never user content.
Tenant/subject attribution on executor.tool.execute, so error queries no longer need a trace-id join against the outer request span.
http.status_code on the plugin.openapi.invoke wrapper span (in addition to the inner span).
Derived message getters for the message-less SDK tagged errors (ToolNotFoundError, ToolBlockedError, PluginNotLoadedError, NoHandlerError, ConnectionNotFoundError, ElicitationDeclinedError, integration lifecycle errors), and formatInvocationCauseMessage now prefers the cause's _tag / structural stringify over [object Object].
An end-to-end telemetry contract scenario (e2e/cloud/telemetry-contract.test.ts): drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real local upstream returning 502 → span batch → OTLP export — and asserts the spans as the suite's trace store actually received them (failure outcome attributes + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, success distinguishable from failure). The prod bug class this guards against (an attribute stamped on a span the exporter never carries, a failure riding the success channel) only manifests at the exported layer, so that's where the contract is pinned. Adds a Telemetry e2e service that queries the suite's OTLP store with arrival polling.

Verification

The e2e scenario passes against the full cloud dev stack (real workerd topology, suite-owned OTLP store) and fails if any stamped attribute, span name, or outcome distinction regresses at the exported layer.
packages/core/sdk, packages/core/execution, packages/plugins/openapi suites green; lint, format:check, typecheck clean.

Stack

Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract #992 👈 current

cloudflare-workers-and-pages · 2026-06-12T22:44:29Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	executor-marketing	`e9dda28`	Commit Preview URL Branch Preview URL	Jun 15 2026, 02:34 AM

cloudflare-workers-and-pages · 2026-06-12T22:44:30Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	executor-cloud	`e9dda28`	Jun 15 2026, 02:34 AM

github-actions · 2026-06-12T22:47:11Z

Cloudflare preview

Torn down — the PR is closed.

pkg-pr-new · 2026-06-12T22:48:31Z

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@992

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@992

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@992

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@992

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@992

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@992

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@992

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@992

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@992

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@992

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@992

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@992

executor

npm i https://pkg.pr.new/executor@992

commit: e9dda28

…rror messages, recording-tracer contracts Expected tool failures (ToolResult.fail) ride the Effect success channel, so the tracer recorded healthy spans while users hit upstream error walls — in a week of prod data, zero of 19.5k plugin.openapi.invoke spans carried http.status_code (annotated inside the inner OpenApi.invoke span, not the wrapper queries target) and 562 of 804 executor.tool.execute error spans had an empty status.message (tagged errors with no message field). - Stamp executor.tool.outcome / error_code / error_status on executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult, plus tenant/subject attribution so queries no longer need a trace-id join against the outer request span. - Annotate http.status_code on the plugin.openapi.invoke wrapper span. - Derive messages for the message-less SDK tagged errors and stop formatInvocationCauseMessage rendering plain objects as [object Object]. - Add a recording Tracer to @executor-js/sdk/testing and telemetry contract tests that pin the span names, attributes, and error statuses production queries depend on.

Replace the recording-tracer unit tests with one e2e scenario that drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream returning 502 → OTLP export — and asserts the spans as the suite's motel store actually received them: outcome/error_code/error_status + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, and ok/fail distinguishable. The unit tests verified span objects in process; the prod bug class this guards against (attribute stamped on a span the exporter never carries, failure riding the success channel) only shows up at the exported layer. Adds a Telemetry e2e service (motel span search with arrival polling), provided when the suite's motel boots (E2E_MOTEL_URL).

- prod-telemetry repo skill: the executor-cloud Axiom dataset layout (attributes.custom JSON map, span names and their attributes including the new outcome fields), working APL recipes for error digests and org attribution, prod-DB and PostHog access notes, and the deploy-canary procedure. - e2e/AGENTS.md: the Telemetry service — scenarios can assert on the spans the target actually exported, with the arrival-polling and fixture-tagging gotchas.

greptile-apps · 2026-06-15T02:38:03Z

Greptile Summary

This PR makes expected tool failures (ToolResult.fail) visible in telemetry by annotating outcome attributes onto spans, fixing three production blind spots audited on 2026-06-12: missing http.status_code on plugin.openapi.invoke wrapper spans, empty status.message on tagged SDK errors, and [object Object] labels from non-Error causes. An end-to-end telemetry contract test pins these fixes at the OTLP-export layer where the silent-failure regression class actually manifests.

Span outcome attributes (executor.tool.outcome, executor.tool.error_code, executor.tool.error_status) are stamped on both executor.tool.execute and mcp.tool.dispatch via a new annotateToolResultOutcome helper; executor.tenant/executor.subject are added to executor.tool.execute so error attribution no longer requires a trace-join.
http.status_code is now annotated on the plugin.openapi.invoke wrapper span (in addition to the inner OpenApi.invoke span) by placing Effect.tap inside Effect.withSpan.
Derived message getters are added to all message-less Schema.TaggedErrorClass errors, and formatInvocationCauseMessage now prefers _tag over String(plainObject) to eliminate [object Object] labels in telemetry.

Confidence Score: 4/5

Safe to merge; all three production gaps are correctly addressed and the e2e contract pins them at the export layer.

The span annotation ordering, Effect tap/withSpan nesting, and skip mechanism are all correct. The one issue is in the new telemetry surface: Effect.promise rather than Effect.tryPromise means a transient network error while polling the motel store becomes an unretryable defect rather than a retried failure — harmless for a local CI motel but worth hardening before the pattern is copied elsewhere.

e2e/src/surfaces/telemetry.ts — the Effect.promise calls for fetch and response.json() should be Effect.tryPromise to keep network errors inside the retry loop.

Important Files Changed

Filename	Overview
packages/core/sdk/src/tool-result.ts	Adds `annotateToolResultOutcome` — stamps outcome/error_code/error_status on the current span; correctly handles ok, fail, and non-ToolResult values; annotation always succeeds (no-ops without a span context).
packages/core/sdk/src/errors.ts	Adds `message` getter overrides to all message-less tagged errors; getters are prototype-level so serialization/encoding is unaffected; coverage is complete across all tagged error classes in the file.
packages/core/sdk/src/executor.ts	Places `annotateToolResultOutcome` via `Effect.tap` inside the `executor.tool.execute` span (correct ordering); adds `executor.tenant`/`executor.subject` span attributes; improves `formatInvocationCauseMessage` to prefer `_tag` over `[object Object]`.
packages/plugins/openapi/src/sdk/invoke.ts	Stamps `http.status_code` on the `plugin.openapi.invoke` wrapper span via `Effect.tap` before `withSpan`, so the annotation runs inside the correct span context.
e2e/cloud/telemetry-contract.test.ts	New e2e telemetry contract test; drives the full production topology to an OTLP store; uses unique random slugs for isolation; upstream HTTP server is properly scoped with `acquireRelease`; assertions cover both outcome classes and tenant attribution.
e2e/src/surfaces/telemetry.ts	New telemetry surface for querying the motel OTLP store; `expectSpan` polls correctly via `Schedule.both`; uses `Effect.promise` instead of `Effect.tryPromise`, which means network errors become defects that bypass the retry loop.
packages/core/execution/src/tool-invoker.ts	Correctly calls `annotateToolResultOutcome` before the `isToolResult` guard so both ToolResult and raw-value outcomes are stamped on the `mcp.tool.dispatch` span.
e2e/src/scenario.ts	Wires `Telemetry` into the test context when `E2E_MOTEL_URL` is set; the existing `missingServices` skip mechanism correctly converts an absent `Telemetry` service into a vitest skip rather than a failure.
e2e/src/services.ts	Adds `Telemetry` Context.Service with key `e2e/telemetry`; pattern matches existing optional services; jsdoc clearly states the availability condition.
packages/core/sdk/src/elicitation.ts	Adds `message` getter to `ElicitationDeclinedError` with clear cancel/decline distinction; consistent with the pattern applied across errors.ts.
packages/core/sdk/src/index.ts	Exports `annotateToolResultOutcome` from the SDK; grouped correctly with the existing tool-result exports.
e2e/setup/cloud.globalsetup.ts	Guards `E2E_MOTEL_URL` behind `if (motel)` so unset env propagation never overwrites a missing motel with an empty string.

Sequence Diagram

sequenceDiagram
    participant C as Caller
    participant E as executor.tool.execute span
    participant D as mcp.tool.dispatch span
    participant O as plugin.openapi.invoke span
    participant U as Upstream HTTP

    C->>E: execute(address, args)
    activate E
    Note over E: annotates tenant/subject at span open
    E->>D: dispatch to sandbox
    activate D
    D->>O: OpenAPI invoke
    activate O
    O->>U: HTTP GET /fail
    U-->>O: 502 Bad Gateway
    Note over O: annotates http.status_code=502
    O-->>D: ToolResult.fail
    deactivate O
    Note over D: annotateToolResultOutcome()
    D-->>E: ToolResult.fail (success channel)
    deactivate D
    Note over E: Effect.tap(annotateToolResultOutcome)
    E-->>C: ToolResult.fail
    deactivate E
    Note over E,O: All three spans carry failure attributes

Comments Outside Diff (1)

e2e/src/surfaces/telemetry.ts, line 476-484 (link)

Effect.promise bypasses retry on network errors

Effect.promise converts rejected promises into Effect defects (the Die channel), which Effect.retry in expectSpan does not catch — only expected failures are retried. If the motel HTTP endpoint is momentarily unreachable (e.g., a transient ECONNREFUSED or a slow response.json() parse), the defect propagates directly through the retry schedule and kills the poll loop immediately. Effect.tryPromise would convert the rejection into a retryable failure instead, giving the full ~20 s window to recover.

_{Reviews (1): Last reviewed commit: "Document the telemetry surfaces for futu..." | Re-trigger Greptile}

RhysSullivan force-pushed the claude/vigilant-driscoll-1a71ea branch from e515a9f to c431459 Compare June 12, 2026 22:44

RhysSullivan marked this pull request as ready for review June 15, 2026 02:28

RhysSullivan added 4 commits June 14, 2026 19:28

Move product-observability plan note out of the repo

e7c6149

RhysSullivan force-pushed the claude/vigilant-driscoll-1a71ea branch from 9c1dc8d to e9dda28 Compare June 15, 2026 02:31

RhysSullivan changed the title ~~Make tool failures visible telemetry: span outcome attributes, derived error messages, tracer contract tests~~ Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract Jun 15, 2026

RhysSullivan merged commit 130d0e9 into main Jun 15, 2026
14 checks passed

RhysSullivan deleted the claude/vigilant-driscoll-1a71ea branch June 15, 2026 02:37

Navigation Menu

Uh oh!

Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract - #992

Make tool failures visible telemetry: span outcome attributes, derived error messages, e2e contract#992
RhysSullivan merged 4 commits into
mainfrom
claude/vigilant-driscoll-1a71ea

RhysSullivan commented Jun 12, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 15, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

RhysSullivan commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Verification

Stack

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cloudflare preview

Uh oh!

pkg-pr-new Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Comments Outside Diff (1)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RhysSullivan commented Jun 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jun 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

pkg-pr-new Bot commented Jun 12, 2026 •

edited

Loading

greptile-apps Bot commented Jun 15, 2026 •

edited

Loading